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Il5 Object Recognition Using a Belief Model," to Bella, et al., filed June 16, 2000, the contents 
I of which are incorporated herein by reference in their entirety. 

'l Background of the Invention 

% a2 0 Field of the Invention 

The present invention relates generally to expert systems and more particularly to 
" recognition systems. 
Related Art 

Conventional image recognition systems are not well suited to recognizing arbitrary 

2 5 objects. The sheer number of possible objects to be recognized, and the infinite variety of 

representations, relations, views and scale give rise to a multitude of intricate problems. In 
addition, the sources of raw image data are quite large. For example, images that may need to 
be analyzed by a company or government agency can be captured from various content 
sources including, e.g., the worldwide web (WWW), newspapers, magazines, flyers, cameras, 

3 0 airplanes, missiles, people, signs, buildings, maps and various video sources such as, e.g., 
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newscasts, documentaries, camcorders, teleconferences, digital and analog video, people, and 
locations. A scaleablc approach in terms of database size and the number of recognizable 
objects is needed to adequately filter and route this data. 

Conventional object recognition solutions have shortcomings. The conventional 
solutions are geared towards recognizing specific objects of interest. The conventional 
solutions do not handle arbitrary images and do not easily scale to larger domains in terms of 
the numbers of objects they recognize. 

What is needed then is a solution that provides adequate scaleability in terms of speed, 
accuracy, and domain size. 



Summary of the Invention 
^ An exemplary embodiment of the present invention is directed to a system, method 

and computer program product for providing an object recognition blackboard system 
architecture. 

r% 5 In an exemplary embodiment of the present invention, the system for recognizing 

(\JjL ( ht b /^[^T objects in content can includ^a blackboard comprising a plurality of experts, and data 
comprising original input data and data created by processing of any of the plurality of 
^jjU<lf)t *^{$~ ex P erts > an£ i a controller operative to control the expert^/a belief model, coupled to the 

controller, comprising a set of beliefs and probabilities associated with each belief of the set 
$JU'lh< of belief^/a belief network, coupled to the controllei^and a relations subsystem, coupled to 

the controller. 

In an exemplary embodiment, the experts can include expert object recognizers 
^\Mt fa ^i^t^T including experts such as^rgT, region identification experts; color region experts; a corner 
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recognizer; a closed curve recognizer; a roof recognizer; a text recognizer; simulated experts; 
microphone recognizer; space suit recognizer; satellite recognizer; a geometric shape 
recognizer; a building recognizer; an egg recognizer; a dice recognizer; a person recognizer; a 
face recognizer; or a product recognizer. 

In an exemplary embodiment, the data can include, relations data; expert status 
data; image subsection data; and the belief model. 

In an exemplary embodiment, the controller can be configured to be operative to 
choose chosen experts from the plurality of experts which are to be executed^o be operative $6( ^ ffi' 
to schedule execution of the chosen expert^ or to be operative to execute the chosen experts. ^ , 0, (p f V ((fif 

In an exemplary embodiment, the blackboard can further include storage for receiving 
an input imag^f or a reporter operative to output results of processing. ' ^' ^( $i 

In an exemplary embodiment, the belief model can include a set of rules deduced from 
a learning system which describes how different classes recognized by the system are related 
to each other spatially and physically. 

In an exemplary embodiment, the belief model is operative to predict existence of a 
shadow object in an image even if there are no specific experts capable of recognizing the 
shadow object. 

In an exemplary embodiment, the belief network is operative to combine beliefs in 
output data output by the experts and probabilities drawn from the belief model into a single 
belief for a given object. 

In an exemplary embodiment, the relations subsystem is operative to determine how 
returned objects returned by the experts are related to each other. 
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an exemplary embodiment, the relations subsystem is operative to determine spatial 



relations. 



In an exemplary embodiment, the spatial relations include types including, e.g., a 
north type, a south type, an east type, a west type, a contains type, a contained by type, or an 
5 adjacent to type. 

In an exemplary embodiment, the relations subsystem is operative to determine 
temporal relations. 

In an exemplary embodiment, the temporal relations include types including: a before 
^ type, an after type, or an exists with type. 

QS 1 0 In an exemplary embodiment, the content can include: video; an image; digitized 

content; or a frame. 

X In an exemplar}' embodiment, the belief model is generated by a learning system, 

rj In an exemplar}^ embodiment, the learning system includes: truth data files for 

U deducing beliefs, probabilities and shadow objects; a learning system controller; or a statistics 
5 space controlled by the controller. 

In an exemplary embodiment, the learning system is operative to assist in integrating 
a new expert wherein tire new expert has been created, encapsulated, compiled, a stub 
function has been added to the blackboard, if output is new has been added to the belief 
model, and a blackboard rule has been added to control when the new expert will be 
2 0 executed. 



In an exemplary embodiment, the belief network is a/a Bayesian Network/a mean 




probability/ or a Dempster-Shafer Network. 
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In an exemplary embodiment, the belief model includes/rules operative to be used to t>j/U 4 $ 0f 
make a determination whether or not one of the experts should be executed by search of the 
belief model to determine whether an adaptable threshold of supporting evidence has been 
exceeded for an execution supportability rule that evaluates outputs of currently executing 
experts. 

In an exemplary embodiment, the belief model is operative to model expected object 
associations, to weigh relative object positions, and to tie a probability or belief value to those 
associations. 

In an exemplary embodiment, the belief network is operative to combine the belief 
model with hypotheses generated by the experts to form belief values for hypothesized 
objects. 

In another exemplary embodiment of the present invention a method of recognizing 
objects is disclosed including^identifying classes of objects specified by a user using a ^ k/V/#f~ 

plurality of cooperative object recognition expert^ achieving higher accuracy from using in f/J l/y/gf 

parallel the plurality of cooperative object recognition experts than is achievable using in 

serial the plurality of cooperative object recognition experts/supporting scaleability of ^ h/v/tfif 

performance including supporting multiple processors^developing a belief model including &A*ri. (zfv/fif 
specifying specified associations among the objects, learning learned associations among the 
objects, representing the specified and learned associations, and forming a belief network 
wherein the belief network is at least one of a Bayesian Network and a Dempster Shafer 

Network^ and deducing shadow objects from the belief model. (H< <t\ & /V^ $f 

In another exemplary embodiment, a method for adding a new expert to a blackboard 
is disclosed including/creating an expert/encapsulating the expert/compiling the expert/ tj/U 4> (Hi (f[~ 
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adding a stub function to a blackboardyiietermining if output of the expert is new and if new, 6j/U „ ^ (#/ 
then adding the output's class to the blackboard, and updating a belief model by providing 
truth data file data to a learning systeiy^and creating a rule to control when the new expert is<^ ^ { ^ 
to be executed when supporting evidence is found to exceed an adaptable threshold. 

In an exemplary embodiment, the system can include a plurality of expert^a belief fytt (#/ ^f/tff 
model^and a controller operative to control processing of the plurality of expert object fyt{ £ ^/^f $\f^ 
recognizers on image. 

The belief model, in an exemplary embodiment, can model expected object 
associations, such as, relative object positions, and can tie a probability or belief value to QMS. Io/h(¥J 
those associations. 

The blackboard controller in an exemplary embodiment can further include a belief 
network operative to combine the belief model with hypotheses generated by plurality of 
expert the object recognizers to form belief values for hypothesized objects. 

The present invention, in an exemplary embodiment, can provide for easy 
modification of the belief model, and due to extensibility of the object recognition 
Blackboard system architecture, can be easily scaled to incorporate additional objects and 
new technologies. 

The present invention can provide various advantages. The present invention 
advantageously recognizes that using a combination of the plurality of expert object 
recognizers more accurately recognizes certain types of objects. 

Further examples of advantages include, object recognition accuracy can be fyt( t fa tofa^ xjff 
gained over conventional expert object recognizers executed alone/performance can be (j\^A t 0, 6/y ( <j^f 
gained using parallel processing over the conventional export object recognizers executed 
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serially^relative ease of extending domain size by integration of new technologies and tyAisfft (* / *f C $f 
objects by a user/and the ability to define and predict scene information and existence of fijj^ $ ^ (^f<ff 
objecrs for which specialized expert object recognizer processing does not exist. 

Further features and advantages of the invention, as well as the structure and operation 
5 of various embodiments of the invention, are described in detail below with reference to the 
accompanying drawings. 

Brief Description of the Drawings 
Ttie foregoing and other features and advantages of the invention will be apparent 
3 0 from th e following, more particular description of a preferred embodiment of the invention, 
y* as illustrated in the accompanying drawings wherein like reference numbers generally 

indicate identical, functionally similar, and/or structurally similar elements. The left most 
l. } digits in the corresponding reference number indicate the drawing in which an element first 
Ci appears. 

P 5 PI G. 1 A depicts an exemplary embodiment of a block diagram of an image object 

recognition blackboard system according to the present invention illustrating data 

instantiation and data flow; 

FIG. IB depicts an exemplary embodiment of a block diagram of an image object 

recognition blackboard system according to the present invention illustrating data flow and 
2 0 control flow; 

FIG. 2 depicts an exemplary embodiment of a block diagram of exemplary city, nest 
and dice scenes illustrative of examples of the present invention; 

FIG. 3 depicts an exemplary embodiment of a block diagram of a learning system of 
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the image object recognition blackboard system according to the present invention; 

FIG. 4 depicts an exemplary embodiment of a block diagram of an exemplary four 
quadrant cartesian .coordinate plane illustrating quadrant overlap for use in performing image 
object recofflition according to the present invention; - , ^^^^^ 6^f^ 
5 FIGs, 5A-5D depict an example of the output of each of the region identification 

experts described according to an exemplary implementation of the present invention; 

FIG. 6 depicts a chart graphing belief versus images for scene prediction for generic 
news images according to an exemplary embodiment of the present invention; 

FIG - 7 depicts a chart graphing belief versus images for scene prediction for space 
=1 0 images according to an exemplary embodiment of the present invention; 
*] FIG * 8 depicts a chart, graphing belief versus image for scene prediction for soccer 

k images according to an exemplary embodiment of the present invention; 
I) FIG - 9 depicts a chart graphing number of shadow objects versus images for number 

of predicted shadow objects compared to number of true shadow objects according to an 
1 5 exemplary embodiment of the present invention; 

FIG. 10 depicts a chart graphing average belief versus image for true shadow object 
predictions against false shadow object predictions according to an exemplary embodiment of 
the present invention; and 

FIG. 1 1 depicts a chart graphing average belief versus image for true shadow object 
2 0 predictions against false shadow object predictions minus scene shadow objects according to 
an exemplary embodiment of the present invention. 
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achieved is encouraging. 

It has been shown that the Blackboard framework of the present invention is capable 
of accepting actual image processing processes as experts, and of utilizing the output of those 
experts. It was possible to create a belief model that was capable of predicting correct scene 
classifications in our test data with a > 0.67 percent accuracy. The belief model was also 
capable of predicting the possible existence of 6 non-scene shadow objects, although the final 
belief values of these object rendered the discrimination between true and false predictions to 
be somewhat ambiguous. 

The results show that even with a less than perfect set of image processing experts and 
an overly simplified belief network, it was possible to combine the output from the experts 
and get useful results. If a more sophisticated belief network was implemented and a more 
specialized set of image processing experts and relation functions used, we believe that the 
ambiguity in non-scene shadow object prediction could be reduced while at the same time 
improving the scene prediction results. 

Generic Ne>vs Scenes ^ 
A total of 33 images of generic news footage were selected and processed. Table 

shows the overall results for each of the images. 
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Space Scenes 

A total of 37 images of space related footage were selected and processed. Table p^T ^j/lA / 0/ ^ 
5 shows the overall results for each of the images. 
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World Cup Soccer Scenes 



A total of 37 images of World Cup soccer footage were selected and processed. Table 
^r^shows the overall results for each of the images. 
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Image IS 
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Belief Model; Definitions 
ists the I 







Obiect Class 


Id 


Person 


i 
\ 


Face 


2 


Tnvt 
1 CXI 


1 

J 


Microphone 




Inset 


< 
J 


Earth 


/r 
0 


Satellite 


/ 


Space Suit 


o 
5 


Soccer Field 


9 


Ball 


10 


Generic Nev/s Scene 


1 1 


Space Scene 


t o 
\l 


Soccer Scene 


13 


Region type 0 




Region type 1 


15 


Region type 2 


16 


Unknown color 


17 


Black 


18 


Blue 


19 


Red 


20 


Green 


21 


Cyan 


22 


Magenta 


23 


Yellow 


24 


White 


25 



ike 0. bM*f~ 



y^W^ Tabie joists the Ids 



associated with the relations used in the Belief Model. 
Table 



Relation Type 


Id 


North 


1 


South 


2 


East 


3 


West 


4 


Contains 


5 


Contained By 


6 


Adjacent To 


7 



10 



Belief Model: Rules 

(belief-model (class 1 1 j (shadow 1 1) (relation 6) (clas$2 13) (shadow2 I) (mean 0.5313) (stddev 0.4990) (prob 0.53 13)) 
1 5 (belief-model (class 1 2) (snadowi I) (relation 1 ) (class 2 1 ) (shadov/2 1 ) (mean 0 5) (stddev 0.5) (prob 0.5)) 

(belief-model (class! 2)(shadowl ^(relation I) fclass2 3) (shi,dow2 0)(mcan 0.8333) (stddev 0.6372) (picb 0.6667)) 
(belief-model (class 1 2) (shadowl 1) (relation 1) (class2 24) (shadow2 0) (mean 1.5) (stddev 1.2583) (prob 0,6667)) 
(belief-model (class) 2) (shadowl 1 ) (relation 3) (class2 20) (shadow2 0) (nvan 1 .5) (stddev 1 .5) (prob 0.6667)) 
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